Docket No. CR-1340 
09/930,322 



REMARKS 

Claims 1-19 are pending in the application. Claims 1-11, 13, and 15-18 are 
rejected. Claims 12, 14, an 19 are objected to. All rejections and objections are 
respectfully traversed. The claims have been amended to clarify what is claimed, 
and not to add any new limitations. 

Claim 4 has been amended to overcome the Examiner's claim objection. In 
essence, a projector is registered with the 3D object because the projector is 
intended to illuminate the 3D object with a projected (corrected) image. 

The Examiner rejected claim 13 under 35 USC 1 12. Claim 12 has been amended to 
overcome the rejection. 

Claims 1, 3-4, 6-11,15-18 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Kitamura (U.S. 6,936,628), in view of Lake et al., (U.S. 
6,593,924 -"Lake") in view of Oosawa (U.S. 6,636,627) in view of Stegmann 
(U.S. 6,593,924). 

Kitamura creates a 3D model from 3D input data. The model can then be edited 
and displayed on a display unit, such as a CRT or CAD/CAM system. The 
Examiner states that "Kitamura discloses a system for animating a 3D physical 
object." This is incorrect, Kitamura, at best, animates the model of the 3D physical 
object as viewed on a display unit (4), see Figures 1, 2 and 4. Display units have 
flat 2D screens, and output images are rendered onto the 2D screen. 
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The claimed invention animates the 3D physical object itself. That is, the output 
images are rendered onto a real 3D physical object. Nowhere does Kitamura 
discuss, disclose, describe, teach, or show a "method for animating a 3D physical 
object." 

The figure on the left below contrast the present invention with Kitamura on the 
right. 



3d object 3d model 3d object 3d model 




2d image 2d image 

on CRT 



Nothing in Kitamura is of any use to make the claimed invention obvious. 
Kitamura is non-analogous art. 

Two specific examples are given in the application, one is a wooden physical 
reproduction of the magnificent Taj Mahal including its signature four minerettes 
and distinctive dome, the other is a wooden replica of a Volkswagen beetle. Both 
mock-up physical objects are painted a neutral white color. The output images are 
rendered on the 3D surfaces of the objects. For example, the car can be made any 
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color pattern with wheels that appear to rotate, the Taj Mahal can reflect night or 
day lighting conditions. Kitamura comes nowhere close to this. Displaying an 
output image so that it appears correctly on a multi-faceted physical object with 
many surfaces at many different angles and curvatures is a much more complicated 
task that merely displaying an image on a CRT. Kitamura does not even consider 
this problem. Kitamura does not have to carefully align his output image with a 
complex shaped 3D real world object during the rendering. 

Lake does not cure the many defects of Kitamura. Like Kitamura, Lake is also only 
concerned with displaying images on a 2D screen. Lake combined with Kitamura 
does not teach the claimed illumination of images onto surfaces of real 3D physical 
objects. Lake operates entirely in a virtual world. Lake does disclose a virtual light 
source and a virtual eye-point. However, the claimed invention does not deal with 
a virtual point of view. Instead, the invention is concerned with a real person 
looking at the 3D physical object while it is being illuminated. As the person 
moves around the object, the invention corrects the output image to reflect the 
current user location, e.g., by adjusting shadows, secular lighting, radiance, etc. 
Thus, as the user walks around the object, the appearance of the object changes 
depending on the user location with respect to the object. This is an entirely 
different problem than that of Lake or Kitamura. In Lake, if the user walks around 
the object where his images are displayed would only see the back of a CRT. 

Oosawa does not correct the deficiencies of Kitamura and Lake. Again, so there is 
no confusion, Oosawa also renders onto a flat display screen. It must be understood 
that the invention renders a model of a 3D physical object back onto the 3D 
physical object itself. Moreover, Oosawa's is concerned with determining the 
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location of a light source from input images so that the location of the light source 
can be properly reflected in the model and output images. 

The invention is concerned with something else, here the virtual light source 
reflects the location of the user and the shape of the surfaces of the 3D physical 
object. For example, as the user walks around the Taj Mahal mock-up, the intensity 
of the lighting can appear to come from the east, if a morning scene is depicted, 
and the lighting can be subdued if it is the evening. The lighting correction as 
claimed has nothing to do with the lighting of the input images as in Oosawa. In 
fact, the physical objects described for the present invention are painted a neutral 
white, so that lighting differences at the different surfaces are minimized. This is in 
exact contrast with Oosawa. 

Stegmann displays a 2D design onto a 3D object. What is claimed is rendering an 
image derived from a 3D model onto a 3D object, which represented by the model. 
Stegmann is totally incompatible with Kitamura, Lake, Oosawa, as well as the 
invention. There is no way that Stegmann can be combined with Kitamura, in 
either direction. Stegmann starts with a 2D design, not a 3D model of a 3D object. 
Kitamura's output images are incompatible with the 3D objects of Stegmann, and 
Stegmann 's output designs as well as his input are incompatible with the 2D 
display screen of Kitamura. 

Nowhere does Stegmann describe any projectors, and certainly not registering a 
projector with a 3D object. The Examiner cites column 1, lines 5-28. This invention 

relates to a Virtual Design System that uses a mathematical 3D model of a real world object, performs an application 
of a design on that model, and which can be combined with an Optical Projection System to visualize the design 
and/or design data on the real world object. Standard 3D-CAD (Computer Aided Design) systems are at present 
widely used or the realization and visualization of designs on objects. This includes the representation of an object 
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from different perspectives, change of light sources, shadowing, rendering, animation, and related features. These 
standard systems represent a real world object internally as a mathematical model, which incorporates the three- 
dimensional data of the object in form of primitives such as lines, polygons and solid bodies, and an instruction set 
of the operations which can be performed to manipulate the data. Application of designs on 3D objects with help of 
these systems is done in the form of rendering techniques. Bitmap patterns are projected on the 3D-object for 
visualization purposes only, this is called texture mapping. For example, a fashion design is evaluated visually by 

trying out different textures and color combinations on different parts of the design." 

Applicants would like the Examiner to point out which word or words in the cited 
section means "registering a projector with the 3D physical object," An optimal 
manual placing of a laser as in Stegmann is not an optical registration process as 
known in the art. Optimal placing is a brute force manual process. Registration is 
automatic and requires the determination of extrinsic and intrinsic parameters, 
projection transformation matrices, homographies and the like. Note, because the 
registration is automatic, the placement of the projector can be approximate, and 
does need to be optimal as in Stegmann. 

The Examiner's statement regarding claim 6 is pure conjecture. No references are 
supplied. Applicants believe that the editing as claimed is novel. 

The Examiner's comment regarding claim 7 and 8 makes no sense, "it would be 
obvious to one skilled in the art to track the location of a moving user because the 
references looks at objects from different viewpoints so knowing the locations 
would be needed when rendering the model." Applicants are not sure what the 
Examiner means by 'references looks'. 

In addition, tracking a user with respect to a 3D model is a non-trivial task. None 
of the references supplied address this problem, since the references do not have 
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the requirement of projecting an image onto a 3D real world object. In addition, the 
invention renders the 3D model according to the location of the user. This also is a 
non-trivial task. Applicants believe this feature is novel, and have not seen this step 
anywhere in the literature. Applicants are not aware that any prior art tracks a user 
with a stereo sensor while modifying output images used to illuminate a 3D real 
world object according to the user location and a model of the object 

Regarding claim 9, since the illumination of a 3D object according to a model of 
the self-same object is novel, it should also be understood that the transformation 
matrices and shading parameters that are applied to the 3D model while projecting 
onto the 3D physical object are novel The prior art has never had to consider this 
problem. The Examiner's statement that this is well known is pure conjecture. If 
this is so well known, Applicants respectfully request a reference where this is 
described. Otherwise this rejection should be withdrawn. Applicants are experts in 
the field of 3D object augmentation and have presented numerous papers at 
conferences on this subject, and thoroughly researched the prior art,. But up to 
now, Applicants are unaware of this having been done before. 

Correcting intensity values according to surfaces of a 3D physical object and a 
location of a user, in any manner, has never been done before. 

With respect to claim 1 1 , the point is that the invention is capable of illuminating 
complex shaped 3D object using a 3D model acquired from the object. All the 
primary prior art references supplied by the Examiner deal with the trivial case of 
projecting onto a trivial flat 2D surface. None of those references project onto a 3D 
physical object that includes arbitrarily shaped surfaces oriented at various angles. 
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With respect to claim 15, again, with all due respect, the Examiner misses the 
point. It is not that the appearance of the 3D object of displayed images on a CRT 
seem to rotate. What is claimed is something entirely different. It is the appearance 
of the physical object itself that appears to rotate. The example given in the 
application is a 3D physical car. The body of the car is made out of cardboard and 
paper painted a neutral white color. Obviously the 3D mock-up is static and has no 
visible color. However, by illuminating different parts of the car in a special way, 
for example, by displaying the images in such a way that the wheels of the car 
appear to rotate, the amazing illusion is that the static white car, becomes a speedy 
red racer. 



As stated above Stegmann starts with an arbitrary 2D design. The claimed 
invention begins with a 3D physical object, and derives a 3D model for the object. 
Then, the 3D model is used to illuminate the very same 3D object from which the 
model was derived, see Figure above. 



Stegmann can be picture as: 




With respect to claim 17, the rejection is improper. The Examiner has not 
considered every element in the entire claim. The point is not that it is well known 
that "the projector could be digital," as stated by Examiner. What is actually 
claimed is "the 3D physical object is illuminated with a digital projector." The 
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Examiner has not addressed the issue that Applicants are illuminating a 3D 
physical object with a digital projector. 



As stated above, Katumari does not animate a physical object by actually 
projecting images derived from a model of the object directly onto the object itself, 
as claimed, instead Katumari animates a 3D model by rendering images onto a 
CRT. Applicants respectfully request the Examiner to appreciate this distinction. 



Now, with respect to the user location, again the Examiner does not appreciate the 
what is of concern is the user location with respect to the projected or rendered 
images on the 3D object itself, i.e., the 'illumination'. Now, in Kitamura and Lake, 
the images are displayed on a CRT. So the user location would be with respect to 
the CRT. Obviously, this makes no sense, since the user will see nothing as the 
user walks around the back of the CRT. Thus, considering the user location is pure 
nonsense. 



In contrast, the claimed invention projects onto the 3D object itself. So now, it does 
make sense to correct the output images as the user location with respect to the 
object changes. 

The Examiner rejected claim 2 under 35 USC 103(a), in view of Davidson (U.S. 
6,516,099). Davidson does not cure the defects of the Kitamura, Lake, Oosawa and 
Stegmann. In the claimed invention, the object is scanned to determine a shape of 
an object so that output images derived from a model of the object can be correctly 
projected back onto the self same object. None of the references show or describe 
these steps. 
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The Examiner rejected claim 2 under 35 USC 103(a), in view of Reinhard et al. 
(U.S. 6,5 16,099). Reinhard does not cure the defects of the Kitamura, Lake, 
Oosawa and Stegmann. Again, with all due respect, the Examiner misses the point. 
The editing in claim 5 changes the desired appearance of the object where the 
output images are rendered. In the cited art, the object where the output images are 
rendered is a CRT or some other display unit. Changing the appearance of the CRT 
makes no sense. Clarification is respectfully requested from the Examiner why 
changing the appearance of the CRT would make the invention obvious. 

Applicants request that the Examiner reexamines the last claim element, 
"illuminating the 3D physical object with the corrected images to give the 
physical object the desired appearance" It is the appearance of the object, and not 
the appearance of the model, as in the prior art, that is the ultimate concern of the 
invention. 

In view of the foregoing, it is respectfully submitted that the application is in 
condition for allowance and an early indication of the same is courteously 
solicited. The Examiner is respectfully requested to contact the undersigned by 
telephone at the below listed telephone number, in order to expedite resolution of 
any remaining issues and further to expedite passage of the application to issue, if 
any further comments, questions or suggestions arise in connection with the 
application. 
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To the extent necessary, a petition for an extension of time under 37 C.F.R. § 1.136 
is hereby made. Please charge any shortage in fees due in connection with the 
filing of this paper, including extension of time fees, to Deposit Account 50-0749 
and please credit any excess fees to such deposit account. 



201 Broadway, 8 th Floor 
Cambridge, MA 02139 
Telephone: (617) 621-7573 
Facsimile: (617) 621-7550 



Respectfully Submitted, 
Mitsubishi Electric 
Research Laboratories, Inc. 




rew J. Curtin 
Registration No. 48,485 
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